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SPECIFICATION 



1. Title of the Invention 

Device for recognizing position of vehicle operator 

2. Claims 

1 Device for optical-electric recognition of a target region of a vehicle operator, 
[said target region including] at least a zone wherein the facial region is normally located, 
said device for recognizing position of vehicle operator comprising: 

light emitting means for directing light onto said target region, [said means] 
disposed at a location to the front of the driver's seat, at a predetermined first angle from 
said recognition target region with respect to the front-to-rear axis of the vehicle; and 

imaging means for receiving reflected light from said target region illuminated by 
the light emitting means, and converting same into an electrical signal, [said means] 
disposed at a location to the front of the driver's seat, at a predetermined second angle 
smaller than said first angle, from said recognition target region with respect to the front- 
to-rear axis of the vehicle. 

3. Detailed Description of the Invention 
(Summary of the Invention) 

The present invention is a device for optical-electric recognition of a target region 
that includes at least the facial region of the driver, wherein a particular arrangement for 
the light emitting means and imaging means is employed in order to assure good 
recognition accuracy. 
(Field of Industrial Utilization) 

The present invention relates to a device for recognizing position of a vehicle 
operator, which captures an image of the driving posture of the vehicle operator in order 
to recognize the position of the vehicle operator within the vehicle cabin. 
(Prior Art) 

Rapid developments in electronic devices in recent years have been accompanied 
by ongoing development of new devices for the purpose of improving vehicle handling, 
comfort, and accident avoidance. One such [kind of device] being contemplated [is one 
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capable of] recognizing the driving position of the vehicle operator, for example, the 
position of the driver's eyes or position of the head, and automatically adjusting the 
position and angle of the rearview mirror, headrest, air vents, steering wheel and so on; or 
of sensing if the driver is falling asleep while driving, by detecting periodic change in the 
position of the driver's eyes or position of the head, and issuing an alert to the driver. In 
order to perform such control, a driving position recognizing device is needed in order to 
recognize the driving posture of the vehicle operator; [such a device] could conceivably 
comprise an image sensing portion, consisting of a camera or the like, for imaging the 
driving posture of the vehicle operator and screening the image, and an image processing 
portion, comprising a microcomputer or the like, for analyzing the screened image by 
means of so-called "image processing" to detect on the image a predetermined target 
position, such as the position of the driver's eyes or head, and calculate the position of 
that portion within the vehicle cabin, enabling [the device] to recognize the position of 
the driver. 

(Problem the Invention Attempts to Solve) 

It is necessary that the recognition device described above by able to accurately 
recognize a target region of the driver, for example, the location of the eyes of the 
location of the nose. 

It is therefore an object of the present invention to provide a device for 
recognizing the position of a vehicle operator, having an arrangement for the light 
emitting means and imaging means capable of recognizing with precision a target region 
of the driver. 

(Means for Solving the Problem) 

The present invention is a device for optical-electric recognition of a target region 
of a vehicle operator, [said target region including] at least a zone wherein the facial 
region is normally located, wherein light emitting means for directing light onto said 
target region are disposed at a location to the front of the driver's seat, at a predetermined 
first angle from said recognition target region with respect to the front-to-rear axis of the 
vehicle; and imaging means for receiving reflected light from said target region 
illuminated by the light emitting means, and converting same into an electrical signal, are 



3 



disposed at a location to the front of the driver's seat, at a predetermined second angle 
smaller than said first angle, from said recognition target region with respect to the front- 
to-rear axis of the vehicle. 
(Operation) 

In the above-described arrangement, imaging means situated at the left front of the 
vehicle cabin in a vehicle with the steering wheel on the right side (in a vehicle with the 
steering wheel on the left side, situated at the right front of the vehicle cabin), in 
conjunction with light emitting means, converts an image of the left side of the vehicle 
operator's face into an electrical signal. As the light emitting means illuminates the left 
side (right side) of the vehicle operator's nose with substantially no illumination of the 
right (left) cheek, the imaging means is able to precisely recognize the shape of the nose. 
The optical axes of the light emitting means and imaging means are not blocked by a 
front seat passenger, and the resultant image can be made to include an anteroposterior 
component of the vehicle operator. 
(Description of the Embodiments) 

A first embodiment of the invention is described hereinbelow with reference to 
the accompanying drawings. 

Fig. 1 is a perspective view showing a vehicle instrument panel and surrounding 
area, having installed therein a driving position recognizing device according to this 
embodiment. Symbol 1 denotes the driver, 2 denotes the driver's seat, and 3 denotes the 
instrument panel. [At a location] on instrument panel 3 diagonally leftward in front of the 
driver's seat 2, i.e., forward of the front passenger seat, are disposed a light emitting 
portion 5 that directs light onto the upper torso of the driver 1 from the left, and an image 
sensing portion 6 that captures the upper torso of the driver 1 as a two-dimensional 
image. 

Here, light emitting portion 5 corresponds to the aforementioned light emitting 
means, and consists of an infrared strobe that emits infrared light so that the driver 1 is 
not blinded by the emitted light; as shown in Fig. 2, the device comprises an infrared 
emitter 5a, a lens 5b for directing the infrared beam from infrared emitter 5a onto a wide 
area of the driver, an infrared filter 5c that passes infrared light while blocking visible 
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light, and a case 5d for housing the infrared emitter 5a, lens 5b, and infrared filter 5c, 
installed at a predetermined location in the instrument panel 3. 

Image sensing portion 6 corresponds to the aforementioned image sensing means; 
as shown in Fig. 3, the device is a solid [-state] camera comprising an infrared filter 6a 
that passes infrared light, a lens 6b having a focal distance of 12.5 mm for example, for 
focusing an image of the upper torso of the driver 1 onto the image plane of a solid 
[-state] imaging element 6e, described later, a liquid crystal shutter 6c for controlling the 
amount of light [entering], a phototransistor 6d for sensing the amount of light passing 
through liquid crystal shutter 6c, used to automatically adjust the aperture of the liquid 
crystal shutter 6c, a MOS solid [-state] imaging element 6e comprising a photodiode array 
and switching circuitry, for converting an image focused onto its image plane through the 
infrared filter 6a, lens 6b, and liquid crystal shutter 6c into an electrical signal by means 
of a switching scan, and case 6f for housing the aforementioned elements, and attached to 
the instrument panel 3 . 

The following description of the overall arrangement of the driving position 
recognizing device of the present embodiment is made with reference to the block 
diagram shown in Fig. 4. 

As shown in the drawing, this recognition device comprises, in addition to the 
aforementioned light emitting portion 5 and image sensing portion 6, an image processing 
portion 8 for processing the image of driver 1 captured by image sensing portion 6, and 
recognizing a location on [the body of] the driver (in this embodiment, the location of the 
nose). This image processing portion 8 comprises an image input portion 10 capable of 
controlling image sensing portion 6 as well as converting the image signal from image 
sensing portion 6 into a digital signal, and temporarily storing the converted digital signal 
as image data; a strobe signal output portion 12 for outputting a strobe signal that causes 
the light emitting portion 5 to emit light; a central processing unit (CPU) 14 for executing 
a series of location recognition processes, namely, detecting the location of a unique point 
in a target region of the facial region of the driver (in this embodiment, the location of the 
tip of the nose) from the A/D converted image data stored in image input portion 10, and 
recognizing the location of the nose of driver 1 from the location of this unique point; 
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read-only memory (ROM) 16 having prestored therein data and a control program for 
execution of the location recognition processes by CPU 14; random access memory 
(RAM) 18 for temporary read/write operations on data used for executing arithmetic 
processes; a bus line 20 interconnecting the above components to enable transfer of image 
signals and control signals therebetween; and a power circuit 22 for supplying power to 
the above components. The image input portion 10 comprises a sync signal generating 
circuit 10a for generating a vertical sync signal and horizontal sync signal for image 
sensing portion 6; an A/D conversion circuit 10b for converting the image signal from 
image sensing portion 6 into image data consisting of a digital signal representing 
brightness of for example, 256 horizontal x 240 vertical pixels per frame; and an image 
data memory circuit 10c capable of storing the preceding two frames of converted image 
data. 

The location recognition process executed by image processing portion 8 is now 
described in greater detail with reference to the flow chart shown in Fig. 5. This process 
is executed when a request for position recognition of driver 1 is input by means of a 
periodic signal, signal from a switch, or the like; as noted, in the present embodiment, 
[the process] recognizes the location of the nose of driver i . 

Once the process illustrated in the drawing is initiated, first, Step 101 is executed, 
to perform an image data readout process for outputting a strobe signal to light emitting 
portion 5 so that image sensing portion 6 senses an image [illuminated by] the strobe of 
light emitting portion 5, and stores the sensed image in image input portion 10, in the 
form of image data G consisting of a digital signal. 

Here, image input portion 10 comprises a sync signal generating circuit 10a, an 
A/D conversion circuit 10b, and an image data memory circuit 10c; since image sensing 
portion 6 can be controlled according to a vertical sync signal and horizontal sync signal 
output by sync signal generating circuit 10a, the image signal output by image sensing 
portion 6 sequentially read out, and the image signal having been converted to a digital 
signal by A/D conversion circuit 10b stored as image data G in image data memory 
circuit 10c, it is a simple matter to execute this image data readout process, simply by 
activating the image input portion 10 so that the light emitting portion 5 emits light in 
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sync with the vertical sync signal output by sync signal generating circuit 10a. That is, as 
shown in Fig. 6, the process involves outputting a strobe signal S to light emitting portion 
5 once a predetermined time interval AT has elapsed after time to at which the vertical 
sync signal rises, causing light emitting portion 5 to emit light during the vertical 
blanking interval Ati, and reading out the image produced by the emitted light as image 
data G. 

As noted, the A/D converted image data G stored in image input portion 10 
represents the image from image sensing portion 6 in terms of brightness of 256 
horizontal x 240 vertical pixels, storing the brightness G (x, y) for each pixel as shown in 
Fig. 7. 

Once the light emitting portion 5, image sensing portion 6, and image input 
portion 10 have been activated in Step 101 to read out image data G in image input 
portion 10, the system proceeds to execute Step 102, in which there is performed a bright 
spot search process involving designation of a predetermined target area in image data G, 
namely, the zone indicated by symbol 30 in Fig. 8, as the nose zone, searching for pixels 
having brightness above a certain level, present at the leftmost side of this zone, and 
placing the locations of these in memory. This process is utilized when detecting the 
position of the nose of driver 1 in the subsequent step 103, to limit the search range for 
the nose; the pixel position calculated [in this process] serves as the base point for 
searching for the nose. This bright spot search process can be carried out using as the 
starting point a pixel situated to the left side of the nose zone 30 shown in Fig. 8, with the 
search proceeding in the vertical direction, placing in memory the locations of pixels 
having brightness above an initial predetermined level. 

Once this bright spot search process has been executed, and locations of pixels 
having brightness above the predetermined level, present to the leftmost side of the nose 
zone, have been calculated, the system in the next step 103 now searches for the nose 
using this position as the base point, and performs a process to determine the location of 
the nose. This process involves examining the degree of correlation of image data in 
proximity to the pixels calculated in the bright spot search, with a predetermined standard 
pattern for a nose like that shown in Fig. 9, and designating the location of the pixel 
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having the highest correlation value as the location where the nose is present, to 
determine the location of the nose. 

Specifically, while [designating] each of a total of 121 pixels [in an area] centered 
on the location G(x, y) of a pixel calculated in Step 102 and [extending] 5 pixels each 
way in the vertical and horizontal directions, as shown in Fig. 11, as the center location m 
(0, 0) in the nose standard pattern shown in Fig. 9, correlation values between image data 
and the standard pattern are computed, and the pixel location having the highest value is 
recognized as the nose location G (hx, hy). The center location m (0, 0) in the nose 
standard pattern shown in Fig. 10 is predetermined as the portion corresponding to the 
nose; correlation values may be calculated using the equation: 

£G(X + i,Y+j)xm(i,j) 
Regarding values for i and j, since the nose standard pattern consists of data [for an area 
extending] 16 [pixels] above, 7 below, 1 to the left, and 6 to the right with respect center 
m (0, 0), [a range of] integers from (-1) to (+6) for i, and of (-16) to (+7) for j, are 
acceptable. 

Since correlation values with respect to center location m (0, 0) of the nose 
standard pattern are calculated for a total of 121 pixels [in an area] centered on the 
location G(x, y) of a pixel calculated in Step 102 and [extending] 5 pixels each way in the 
vertical and horizontal directions as shown in Fig. 1 1, the values of X and Y in the 
preceding equation will be integers [in the range] ±5 for X and ±5 for Y; correlation 
values for the maximum of 121 combinations of X and Y are calculated, and the pixel G 
(X, Y) having the greatest value in the image data is designated as the nose location G 
(hx, hy). 

As will be apparent from the preceding description, in the vehicle operator 
position recognition device of this embodiment, a standard pattern for the target region is 
employed during recognition of the target region, whereby the driving position may be 
recognized more accurately, without erroneous recognition of the location of the target 
region. 

A second embodiment of the invention is now described with reference to the 
accompanying drawings. 
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In this [second] embodiment, based on the location of the nose of the driver 1 
recognized [in the manner described in] Embodiment 1, the location of a unique point in 
a target region excluding the nose of the driver 1 (in this example, the center of an eye) 
and the location of a target region (in this example, the eye) are recognized. 

The location recognition process executed in this embodiment is now described in 
greater detail with reference to the flow chart shown in Fig. 11. Step 201, Step 202, and 
Step 203 in this embodiment are analogous to Step 101, Step 102, and Step 103 in 
Embodiment 1, and therefore will not be described again. 

Once the nose location G (hx, hY) has been ascertained by sequentially executing 
Step 201, Step 202, and Step 203, the system proceeds to Step 104, in which the location 
of the eye is now calculated on the basis of the nose location G (hx, hy). Here, when 
calculating the location of the eye, a pixel location G (hx+12, hy -13) 12 pixels to the 
right of and 13 pixels above the nose location G (hx, hy) is calculated, and the degree of 
correlation of the image data to a standard pattern for the eye predetermined for the 
vicinity of this location, shown in Fig. 12, is examined to determine the eye location 
G(mx, my). The determination process is analogous to the nose position determination 
process in Step 103 of Embodiment 1, and will not be described again here. 

Next, in Step 205, on the basis of the eye location G (mx, my) in image data G 
calculated in the preceding Step 204, there is executed a process for computing the three- 
dimensional location of the eye of driver 1 within the vehicle cabin, on the assumption 
that there is no left or right movement of the driver 1. Specifically, this process which, on 
the assumption that there is no movement of the driver 1 in the left/right (Z) direction 
shown in Fig. 13, posits an experimentally-derived coordinate system based on a 
reference location P for the eye (left eye) of the driver in the vehicle and calculates a 
three-dimensional location M of the eye (left eye) of the driver in terms of the coordinates 
(X, Y, O), may be carried out readily by means of pre-storing in memory the location G 
(px, py) of a reference point P in the image data, and calculating deviation of the eye 
position G (mx, my) from this reference location G (px, py), since the image sensing 
portion 5 is fixed. 
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As will be apparent from the preceding description, in the vehicle operator 
position recognition device of this embodiment, a standard pattern for the target region is 
employed during recognition of the target region, whereby the driving position may be 
recognized more accurately, without erroneous recognition of the location of the target 
region. 

In this [second] embodiment, the location of the driver's eye is not calculated 
directly from image data, but rather is detected by first detecting the nose, with the nose 
designated as the target region and the location of the tip of the nose projecting out from 
the face -which is the easiest region to detect in the image data- as the unique location in 
the target region; and then with the eye designated as the target region and the center of 
the eye as the unique location in the target region, detecting the location of the eye based 
on the previously calculated location of the nose, whereby the driving position (i.e. 
location of the left eye) can be recognized readily. 

In the present embodiment, the location of the eye (more precisely, the location of 
the left eye) of the driver 1 is recognized by way of the driving position of the vehicle 
operator; however, a three dimensional location could instead be recognized using, for 
example, the location of the nose of the driver 1 as the driving position, so that the most 
appropriate driving position may be recognized with reference to processes occurring 
after the position of the vehicle operator has been recognized, namely, the process of 
automatically adjusting the rearview mirror, automatically adjusting steering wheel angle, 
and the like. Recognition of the location of the eye of the driver 1 may be utilized in 
automatically adjusting the rearview mirror, sensing if the driver is falling asleep, and so 
on. 

In Embodiments 1 and 2, light emitting portion 5 corresponds to the light emitting 
means, and image sensing portion 6 to the imaging means, with the imaging means 
consisting of an image sensing portion 6 that in the preceding examples comprises a MOS 
solid-state imaging element; however, a CCD or other imaging element employed in so- 
called solid-state camera could be used instead, or a camera tube could be used. 

In Embodiments 1 and 2, a single image sensing portion 6 is provided, and the 
driving position of the driver is recognized using image data of the driver taken from one 



10 



direction, on the assumption that there is no left or right movement of the driver; 
however, image sensing portions could instead be disposed at two different locations, 
using the principle of triangulation to recognize the driving position of the driver, which 
would enable left or right movement of the driver to be detected as well, affording greater 
accuracy in recognition of the driver's 3-dimensional position. 

In Embodiments 1 and 2, an infrared strobe is employed as light emitting means 
II, but continuous light such as an incandescent bulb could be used instead. 

In Embodiments 1 and 2, the bright spot search process has as its starting point the 
upper left edge [of the image], with the search proceeding in the vertical direction; 
however, a search priority ranking could be assigned to all pixels in the nose zone, and 
searched according to this ranking. 

In Embodiments 1 and 2, the nose zone is rectangular, but could instead be round 
or triangular. In this case, effective searches can be carried out by concomitant use of the 
aforementioned method of assigning search priority rankings to pixels. 

Image data obtained from the imaging means, rather than being processed directly 
. for position recognition, could instead be subjected to an edge sensing process or 
horizontal differentiation process, and position recognition performed on the resultant 
image data, [which has the advantage that] the more distinct features of the image data 
afford greater accuracy in position recognition. 
4. Brief Description of the Drawings 

Figs. 1 to 13 illustrate the embodiments of the invention, wherein Fig. 1 is a 
perspective view of an instrument panel incorporating an embodiment of the invention; 
Fig. 2 is an illustration of light emitting portion 5; Fig. 3 is an illustration of image 
sensing portion 6; Fig. 4 is an overall schematic of the driving position recognition 
device; Fig. 5 is a flow chart showing operation of image processing portion 8 of 
Embodiment 1 ; Fig. 6 is a timing chart of the image data G readout process; Fig. 7 is an 
illustration of illuminated image data G stored in image input portion 10; Fig. 8 shows an 
image represented by image data G; Fig. 9 shows data representing a standard pattern for 
a nose; Fig. 10 illustrates a nose location sensing process; Fig. 1 1 is a flow chart showing 
operation of image processing portion 8 of Embodiment 2; Fig. 12 shows data 
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representing a standard pattern for an eye; and Fig. 13 describes the three-dimensional 
location of an eye, wherein (a) is a plan view of driver 1 and (b) is a side view of driver 1. 

5 ... light emitting portion, 6 ... image sensing portion, 8 ... image processing 
portion, 10 ... image input portion, 14 ... CPU 

Agent: ADACffl Tsutomu, Patent Attorney 
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Fig. 2 
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Key: 5 light emitting portion, 6 image sensing portion, 8 image processing portion, 10 
image input portion, 10a sync signal generator circuit, 10b A/D conversion circuit, 10c 
image data memory circuit, 22 power circuit 
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Fig. 10 
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204 eye location G(mx, my) search 

205 eye 3-dimensional location recognition 



Fig. 12 
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Fig. 13 (a) & (b) 
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